With the release of high performance processors such as Intel Corporation's Pentium II and AMD's K7, modern operating systems can now take advantage of the performance gains with the use of the SYSENTER/SYSCALL/SYSEXIT/SYSRET instructions. These instructions use a set of internal registers in the processor, such as the SYSENTER_EIP register (e.g., Pentium II), which holds the code segment, stack pointer and the instruction pointer to use on entry to Ring 0 (Kernel mode) from Ring 3 (User mode subsystems). This user-kernel transition mechanism makes the operating system faster by making the ring transitions more efficient.
However, this mechanism is vulnerable to hijacking. In particular, there is a class of rootkit applications and exploit programs that manipulates with the critical registers of SYSENTER (Intel) and SYSCALL (AMD) instructions in order to take control over a computer system, and is able to monitor all system functions of user mode applications as they happen. These malicious programs do not need to alter any other structures in the target system's memory, or manipulate with the page table. Thus, their functionality is hidden from conventional intrusion detection programs, and therefore more difficult to notice on the system.
What is needed, therefore, are techniques for detecting manipulations of user-kernel transition registers (such as Intel's SYSENTER and AMD's SYSCALL registers) in order to detect potentially malicious or otherwise undesirable actions.